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The purpose of the Review of e-Learning in Canada, conducted by Abrami and associates 
(which I refer to as 'the team'), was to provide evidence and gaps on the topic of e- 
learning, and based on this data provide promising directions for e-learning. Funded by the 
Canadian Council on Learning, the stated objectives of the review were to: 1) identify and 
verify through research the most effective practices and procedures to promote learning; 
2) identify major gaps in our knowledge and understanding of e-learning; and 3) identify 
the most promising lines of inquiry for addressing those gaps. The review covers the 
literature from 2000 forward with an attempt to focus on Canadian literature, in both 
official languages. Using an Argument Catalogue codebook a number of classes were 
identified by the researchers upon which the review was framed. The documents were also 
coded to provide data on outcomes of the e-learning research from the following 
perspectives: general public opinion, practitioner literature, policy documents, scholarly 
reviews, and primary Canadian research. 

Did the team achieve their objectives? And do their outcomes provide us with promising 
directions for e-learning? This commentary will begin by deconstructing the activities of 
this review and conclude with comments on how well the objectives of this review were 
met. 

Introduction 

The introduction begins with the following explanation of the term e-learning: 

E-leaming has become the general tenn encompassing the application of computer technologies to education, 
whether it occurs in face-to-face classrooms, in blended and hybrid courses, in mediated distance education contexts 
or in online learning environments. The Canadian Council on Learning (CCL) defines e-leaming as the development 
of knowledge and skills through the use of infonnation and communication technologies (ICTs) particularly to 
support interactions for learning—interactions with content, with learning activities and tools, and with other 
people. It is not merely content-related, not limited to a particular technology and can be a component of blended or 
hybrid learning. (Rossiter, 2002; also 2005 in an address at the CCL Workshop on E-Learning). The CCL recognizes 
the breadth of the concept of learning through electronic means, as well as its growing pervasiveness in Canadian 
institutions of fonnal learning, at the level of elementary and secondary schools and of colleges, and universities, as 
well as in early childhood and health related learning, (p.9) 


The term e-learning, when used in everyday language, is rich in multiple meanings and, as 
such, beginning with an encompassing definition is essential. Whether the reader agrees or 




disagrees with the team's (or CCL's) description of the term e-learning, establishing 
meaning early in the introduction is essential so readers are operating with the same set of 
understandings. 

Following the definition of the term e-learning is a nicely constructed overview of e- 
learning from both enthusiasts and sceptics. It is also here that the biases and undeclared 
assumptions of the research team begin to emerge. The team cites an article 
fromA laclean's magazine entitled "How computers make our kids stupid". They note that 
this article, which is one person's opinion, criticises the use of technologies. They observe 
further that this expose does not provide an accurate, complete or balanced portrait of the 
use of technologies in the learning process, making the point that this reporter's personal 
perceptions of technologies in the classroom are unfounded. While not disagreeing with the 
team's criticism of Maclean's article, it is certainly possible to find many anecdotes and 
opinion stories advocating the benefits of technologies—which have also been compelling, 
and generating instant and widespread reaction. Although the point the team is trying to 
illustrate with their Maclean's example is that researchers have to "sift through the 
accumulated evidence to form an accurate picture about scientific knowledge" (p. 10)—it is 
revealing that they have selected an opinion story that takes a critical perspective of 
technology to make this point. Selection of this example thinly veils a bias 1) toward 
positive uses of technologies and 2) against personal opinions as a valid form of research. 
I will re-visit this latter point in the closing section of this commentary. 

Approaches and Methodology 

This section provides a detailed explanation of the review process. The study is well 
grounded in a framework for the review process, based on an analysis of news media 
practitioner opinions and policy documents—which the team refers to as their "Argument 
Catalogue". The team articulated the standards upon which they strove to achieve as: 

• A comprehensive review of current documents 

• A review which is repeatable through the use of a transparent review methodology 

• To insure the outcomes of the review are clear, useful, and credible 

• A methodology that is precise, ensuring accurate conclusions 

Did the team achieve these standards? Based on the document provided to me, the 
literature reviewed was current. Was it comprehensive? Based on the team's working 
paradigm, the review was as comprehensive as could be expected given time constraints 
put on the team. On the second standard, was the process for the sources located 
described in sufficient detail to be repeatable? The team has provided a thorough account 
of their activities, including where the sources were gathered from, the criteria for inclusion 
(or not) and how the data were analysed. Though, it should be noted that the links on the 
CSLP website which provides access to the complete citations and Argument Catalogue 
codebook was not operational (404 error). It was also stated by the team that there would 
be emphasis on both French and English sources. However, in the report I counted only 
three French sources, all of which were from La Presse —a privately-owned daily French 
language newspaper aimed at a middle-class readership. 



Results 


In regard to the team's standard on the use of a precise methodology, did they ensure 
accurate results? The selective summary (sample of document conclusions) of results 
brings forward an interesting overview and, indeed, provides us with a rough sketch of the 
evidence and gaps in e-learning within each category. The team has described their 
findings in a concise, yet encompassing manner—while also acknowledging the difficulties 
and challenges encountered with such reviews. The sample conclusions provided in the 
summary included many important documents and research that has had an impact on e- 
learning in Canada. 

Less clear to determine is whether the team achieved this standard in regard to the theme 
areas. On the first part of this standard, I found the methodology to be, as claimed, 
precise, well articulated, and transparent. The team leaders are well known and highly 
regarded for their expertise on literature reviews and this study, consistent with their prior 
work, was no exception. What is less clear is whether the conclusions are accurate. In 
particular, the report provides a rationale for the categorization of the impacts of e- 
learning. Unfortunately, the report does not specify delimitations for these impacts. For 
example, achievement is one of the areas identified as an important impact on e-learning. 
But what does achievement mean? Does it mean successful completion of a course? Or a 
program? Does it mean achieving better grades than other forms of learning? If 
achievement is defined as better grades, what kinds of learning outcomes were 
researched? For example, how is achievement with e-learning different from surface 
learning versus deep learning (e.g., Biggs, 1999; Entwhistle & Ramsden, 1983; Prosser & 
Trigwell, 1991; Trigwell & Prosser, 1991)? Or was achievement based on learning domains 
(e.g., Bloom, 1956; Gagne, 1965)? If so, how is achievement different with e-learning in 
the cognitive domain versus the affective domain versus the psychomotor domain? Or 
does e-learning impact all of these domains equally? Does e-learning impact achievement 
equally across the disciplines? Extensive research conducted by Donald (2002), for 
example, on learning to think across the disciplines revealed that there are significant 
differences in thinking, validation processes and learning activities between disciplines. 
Donald's research shows that the validation process for English involves critiquing other's 
claims, peer reviews and testing the parts against the whole, while psychology uses inter¬ 
rater reliability and empirical testing. Given the significant differences between these 
disciplines, it would be useful to know if achievement with e-learning between disciplines is 
the same. 

Also missing in the review on achievement are research findings which have revealed 
students infrequently engage in the communicative processes that comprise critical 
discourse—an essential component of achievement as it relates to higher levels of learning 
(see for examples: Aviv, Zippy, Ravid & Geva, 2003; Bonk & Cunningham, 1998; Bullen, 
1999; Davis & Rouzie, 2002; De Laat, 2001; Garrison, Anderson & Archer, 
2001;Gunawardena, Carabajal & Lowe, 2001; Gunawardena, Lowe & Anderson, 1997; 
Jeong, 2004; Kanuka, 2005; Kanuka & Anderson, 1998; Lopez-Islas, 2001; McKlin, 



Harmon, Evans & Jones, 2002; McLaughlin & Luca, 2000; Meyer, 2003; Nussbaum, Hartley, 
Sinatra, Reynolds & Bendixen, 2002; Pawan, Paulus, Yalcin & Chang, 2003; Pena-Shaff, 
2005; Pena-Shaff, Martin, & Gay, 2001; Pena-Shaff & Nicholls, 2004; Rourke, 2005; Rovai 
& Barnum, 2003; Thomas, 2002; Vaughan & Garrison, 2005; Veerman, Andriessen, & 
Kanselaar, 2000; Wilson, et al., 2003; Yakimovicz & Murphy, 1995). Research conducted by 
Angeli, Valanides, and Bonk (2003) is representative of many of these studies' 
conclusions: "students primarily share personal experiences amongst themselves, and 
their responses appeared to be subjective and naive at times. Students' discourse was 
also extremely conversational and opinionated and showed little evidence of critical 
thinking" (p. 40). 

It has been difficult for most of us concerned with e-learning in higher education to ignore 
these disappointing results—and, yet, these findings have not been reflected in the team's 
review of the literature. Indeed, a conclusion by the team is that "online technologies 
facilitate the development of higher-order critical thinking; providing great potential for 
educative dialogues" (p. 24). Canadian researchers (e.g., Garrison, Anderson, & Archer, 
2001; Kanuka, 2005; Kanuka & Anderson, 1998; Kanuka, Rourke & Laflamme, in press, 
Rourke, 2005; Vaughan & Garrison, 2005) have had research results that do not support 
this conclusion. This inconsistency is, probably, because the literature review conducted by 
the team was carried out using keyword searches, frequency counts and statistical 
analysis for significance—which I assume (as I cannot access the website for a complete 
set of references) included the well known, and well documented, findings that knowledge 
acquisition (or surface learning) in computer-based courses is significantly better, or not 
significantly different (e.g. Pascarella &Terenzini, 2005; Russell, 1999), and combined with 
the more recent, and not yet as well documented, research on higher levels of learning (or 
deep learning). Such findings, in part, provide a misleading conclusion that e-learning has 
a positive impact on achievement. I state with intent 'in part', as these findings may 
reflect achievement with respect to knowledge acquisition, but not with respect to certain 
aspects of higher levels of learning, such as critical discourse. Most reviews of the research 
acknowledge these kinds of complexities. Pascarella and Terenzini, for example, qualify 
their review of the literature in a number of areas; they observe that engagement is 
essential to their findings: 

A student’s coursework and classroom experiences shape both the nature and extent of his or her acquisition of 
subj ect matter knowledge and academic skills ... what the student does to exploit the academic opportunities 
provided by the institution may have an equal, if not greater, influence ... other things being equal, the more the 
student is psychologically engaged in activities and tasks that reinforce the fonnal academic experience, the more 
he or she will leam. (p. 119) 

Though the team does acknowledge that "there is variability associated with the predictors 
and outcomes [and] ... the quality and quantity of evidence does not allow us to conclude 
with certainty the factors which impact on e-learning" (p. 43). Nevertheless, the team has 
provided us with conclusions. The problems I have identified with the conclusions made by 
the team on achievement can be applied to each of the impacts investigated. How, for 
example, was the construct of motivation operationalized? Were the sources reviewed 
related to Houle's (1961) framework based on activity-oriented (social) motivation, 



learning orientated motivation or goal oriented motivation? Or was it based on Tough's 
(1979) motivational framework for adults which suggests we are motivated to learn (or 
not) based on self-esteem, pleasing and impressing others, or other certain satisfactions? 
As with achievement, does e-learning impact each of these aspects of motivation equally? 
And can we apply the results to all of the theme areas (adult education, early childhood 
education, elementary/secondary education, postsecondary education, health and 
learning) equally? 

Perhaps the most problematic of these constructs, for me, was 
th einteractivity/communication impact. The difficulty I had with this impact was twofold. 
The first troubling aspect was trying to understand why the team placed interactivity in the 
same category as communication. Interacting with an automated computer-generated 
response, for example, is not the same as communicating with one's peers or instructors. 
Secondly, we know from the extensive research in the field of communication that different 
communication technologies have different impacts on how we communicate. Given 
that the most critical aspect of the teaching-learning process is effective communication, 
understanding interpersonal communication with networked communication technologies is 
worthy of a careful review and a thorough explanation of how we communicate with 
technologies. We know, for example, that communication is a complex multimodal process 
that involves not only speech, but also gaze, gesture, and facial expressions (Clark, 1996; 
Clark & Brennan, 1991). Research in this area has given rise to the belief that multimodal 
technologies (such as video conferencing that provides both speech and vision) provide 
more effective communication than single mode technologies (such as audio conferencing 
or e-mail and listservs). A review of this research shows that speech alone can be as 
effective as speech plus video, under certain circumstances speech can be as effective as 
face-to-face communication, and video is not significantly different from speech 
communication (Reid, 1977; Whittaker, 2003). Some research has even revealed that 
adding visual information may impair critical aspects of spoken communication (Anderson, 
et al., 2000; Whittaker & O'Conaill, 1997). Further, there is evidence from the research on 
communication media which indicates that audio systems (e.g., Elluminate, Centra), and 
audio and video systems (e.g., Web cams) can provide more effective interpersonal 
interactions than text-based communication systems alone (Collett, Kanuka, Blanchette & 
Goodale., 1999). Though, this research should be interpreted with caution, as low quality 
video systems (e.g., discontiguous visual and audio transmissions) may provide 
distractions to a point where the communication process and the quality can be severely 
eroded (Whittaker, 1995; Whittaker, 2003). 

Some research within the field of communication has also focused on aspects of media 
richness and/or the effects of filtered cues. Results of this research suggest that different 
communication media affect groups largely through differential transmission of social 
context cues (or paralinguistic cues). Text-based computer mediated communication is 
considered to be 'social cueing poor' as it limits the exchange of paralinguistic and 
interpersonal cues (e.g., age, sex, physical appearance) and other physical surroundings. 
Social cueing is an important aspect that facilitates and regulates interpersonal interaction, 



information and monitors feedback (Straus, 1997). Reductions in social cues through the 
use of reduced-channel media (e.g., text-based communication tools) disrupts the flow of 
communication causing difficulty following and understanding discussions (Straus & 
McGrath, 1994), which can result in diminishing the intensity of interpersonal interactions 
and social connectedness—as well as increasing a sense of anonymity and feelings of 
depersonalization (Straus, 1997). In a depersonalized context, in turn, there can be 
reduced motivation to share personal information and/or inquire about others, as well as 
reduced expressive communication (Hiltz, Johnson & Turoff, 1986; McGuire, Kiesler & 
Siegel, 1987). Explanations for these results tend to revolve around the belief that the 
time and effort required to type versus speak results in considerably less communication in 
text-based discussions than face-to-face—in addition to difficulties in following and 
understanding the text without supplementary social cues, adding to the cognitive 
workload (Straus, 1997). 

This kind full descriptive examination on what the literature tells us about communication 
technologies illustrates this construct is complex, with implications for educators on not 
only the kind of communication technologies, but also the contexts—in contrast to the 
statistical analysis and numerical tables provided by the team which tells us if the 
aggregated results are significantly better, or not. Consequently, it is difficult to know if 
their analysis is accurate because each of the impacts investigated (achievement, 
motivation/satisfaction, interactivity/communication, social demands, attrition/retention, 
learning flexibility, cost) are complex constructs that vary in effectiveness under different 
circumstances. As the team has acknowledged, while the literature points to the positive 
impacts of e-learning on achievement, motivation, communication, learning flexibility and 
meeting social demands, this "consensual evidence alone does not reveal what accounts 
for these positive impressions" (p. 46). 

Perhaps even more important than the absence of boundaries on the impacts, this report 
does not include the consequences of the increased use of e-learning technologies within 
each of the impacts reviewed. Following is a quote from a participant in a recent study I 
conducted, which illustrates why looking at the consequences is important: 

Let’s just think about this as we might about Pfizer. What might they say or do to ascertain their success [or positive 
impact]? Well they might say that 90% of their patients who’ve taken their drug have recovered from chronic renal 
failure. However, unreported is the 97% who got brain cancer from the drug ... We can only say there is no 
significant difference in outcomes if we only look at the apparent production of apparent knowledge outcomes. So, 
like Pfizer, we can produce data that makes us feel good, like the number of students enrolled in e-leaming ... and 
that could then be a justification for more of this ... but it wouldn’t indicate that it is good and what the 
consequences are. It just indicates that a lot of people are prepared to put a lot of money and time into it. 

This quote illustrates the point that philosophers of technologies have argued for decades 
(e.g., Chandler, 1996; Ihde, 1979; Winner, 1993), that research on technologies has not, 
and currently does not, investigate the consequences. A technology is designed to serve a 
specific purpose (intentionality), and as such, it amplifies that aspect (selectivity) of our 
use. But in the process it also unavoidably reduces other aspects of our experience. For 
example, we use a telephone to increase our ability to communicate with others, achieved 



through the ability to gain greater independence of place when communicating verbally 
(amplification). But the experience is less real, as a consequence, than person-to-person 
communication in that we have a loss of many paralinguistic cues (reduction). How much 
this loss matters to us depends on whether such consequences are in harmony with our 
overall intentions, as side effects can, of course, be positive as well as negative. In this 
respect, Winner (1993) has argued that technological artefacts may embody affirmation, 
but may also become a betrayal: "The same devices that have brought wonderful 
conveniences in transportation and communication have also tended to erode community. 
In the maxim of theologian Richard Penniman, 'They got what they wanted, but they lost 
what they had' " (p. 371). Winner maintains that the interesting research questions have 
little at all to do with any alleged self-generating properties of modern technology; rather, 
they have to do with the often-painful ironies of technical choice. Noticeably absent in this 
review are often-painful ironies, or consequences, of the use of e-learning technologies. 

Discussion 

The report concludes with a list of what is currently known about e-learning. I found it 
interesting to read in the introduction of this report a statement by the team that "it is a 
shame to attempt innovation and not be able to tell why it works or doesn't work" (p. 3), 
and yet this is exactly what the team has provided as a summary conclusion "What [bold 
added] We Know: The Evidence". Indeed, it is a shame these summary messages do not 
tell us why it works, or doesn't work, and more importantly under what conditions e- 
learning is most effective, and when it isn't, as well as the consequences. Though, to their 
credit, the team does acknowledge this problem in their discussion: "this review ... does 
not readily present us with evidence of best practices and "what works" in e-learning" (p. 
50). However, if we go back to page 11, we can see that a primary objective of the review 
was to "identify the most effective practices and procedures to promote learning." On this 
point, I can only conclude that the team did not achieve a central objective of this 
literature review. 

Concluding comments 

The methodological approach that we use in our research activities is reflective of our 
epistemological assumptions, and reveals the nature of knowledge we value, in particular 
its foundations, scope, and validity. For this review, the team took a systematic 
compilation of current literature and created an Argument Catalogue codebook. 

While claiming to examine data from a number of perspectives, the team noted (twice) 
that: 


We found that over half of the studies conducted in Canada are qualitative in nature, with the other nearly half split 
between surveys and quantitative studies (correlational and experimental). When we looked at the nature of the 
research designs, again, 51% are qualitative case studies and 15.8% are experimental or quasi-experimental studies. 
It seems that studies that can help us understand “what works” in e-leaming settings are underrepresented in the 
Canadian research literature, (p.l; p. 48) 


Clarification about what the team means by collecting data from a number of perspectives 



is needed. In particular, what is being implied by the team in the above quote about 
qualitative research—in particular the reference to the under representation of 'what 
works'—is puzzling. Specifically, while it would seem we can conclude that the team 
believes that qualitative research tells us nothing in regard to what works they also make 
mention that: 

We fully believe that giving voice to the other, often not considered, sources of evidence, whether they are mere 
opinion, based on practical experience or derived from empirical research is important in developing a complete 
portrait of a field which touches Canadians at so many different levels, and which requires such a substantial 
investment in human and material resources, (p. 57) 

Without further explanation, these statements by the team contradict each other, leaving 
me to wonder what it is that they considered reliable sources for their review. A careful 
read, however, reveals the team's actual views on this point. In the introduction the team 
clearly dismissed the Maclean's article, based on the fact that it was "one reporter's 
attempt to weave together personal anecdotes, stories, and observations" (p. 10). 
TheMaclean's article, which is 'merely one person's opinion', was criticised for lacking 
evidence and being inaccurate, incomplete, and unbalanced. As such, while the team 
'says': "We fully believe that giving voice to the other, often not considered, sources of 
evidence, whether they are mere opinion, based on practical experience or derived from 
empirical research is important in developing a complete portrait of a field", in fact, they 
do not. Rather, it appears they give a privileged voice to the "15.8% experimental or 
quasi-experimental studies" which, according to the team, is the only kind of research 
which tells us "what works". 

These kinds of inconsistencies could be easily be avoided if the team had articulated their 
research assumptions at the beginning of the report and stayed true to their assumptions, 
rather than being politically correct by giving false piety to the diversity of voices. As 
individuals, each of us holds a set of assumptions about how we believe knowledge is 
created and, based on our assumptions, we overlay a framework—or a working research 
design. Based on the research design for this literature review, it is apparent that the team 
is working on the assumption that, given a large enough sample and conducting statistical 
analysis for significance, we can make predictions about e-learning. 

Perhaps more importantly, rather than lamenting over the number of qualitative studies, in 
particular, case studies, the team might have asked the question: Why is so much of the 
e-learning research approached from the qualitative paradigm? A good place to begin 
responding to this question is to acknowledge that despite the polemic nature of 
th eMaclean's article, the e-learning research community must face the reality that our 
efforts have failed to provide adequate guidance for developers and practitioners. As 
Reeves has noted on several occasions (1995; 1999): 

Much of the research in IT is grounded in a “realist” philosophy of science, i.e., conducted under the assumption that 
education is part of an objective reality governed by natural laws and therefore can be studied in a manner similar to 
other natural sciences such as chemistry and biology. If this assumption about the nature of the phenomena we study 
is erroneous (and I believe it is), then we inevitably ask the wrong questions in our research. Further, even if there are 
underlying laws that influence learning, the complexity inherent in these laws may defy our ability to perceive, 
much less control, them (Casti, 1994). Cronbach (1975) pointed out two decades ago, our empirical research may be 



doomed to failure because we simply cannot pile up generalizations fast enough to adapt our instructional treatments 
to the myriad of variables inherent in any given instance of instruction. 15) 


Many e-learning researchers conduct their research based on the assumptions that guide 
natural laws (and that includes this team) and, therefore, also assume e-learning can be 
studied with systematic and rule-governed methodologies. However, this type of research 
has resulted in a large body of knowledge in e-learning which has little social relevance 
due precisely to the fact that it does not reflect the real world messiness of the everyday 
problems that face e-learning practitioners and, therefore, makes little contribution to how 
we can make e-learning better. In this regard, Reeves (1995) argues for 'socially 
responsible' research—or research that aims to make education better, by finding practical 
solutions to the everyday problems faced by e-learning practitioners. Of course all good 
research is time consuming and hard to do, but it takes more than hard work to conduct 
socially responsible research; it takes a desire to discover what the everyday problems 
are for e-learning practitioners and a recognition that we live in a continually evolving 
world, comprised of human beings who are complicated, resulting in 'messy' environments 
—or what Schon (1983) referred to as "the swampy lowlands" of professional practice. A 
key element to generating socially responsible research is the use of more emergent 
methodology that can effectively address the 'real world' messiness that e-learning 
practitioners encounter routinely. On this point, Shulman (1997; see also Cronbach & 
Suppes, 1969), similar to Reeves (1995), argues that "disciplined inquiry does not 
necessarily follow well-established, formal procedures. Some of the most excellent inquiry 
is free-ranging and speculative in its initial stages, trying what might seem to be bizarre 
combinations of ideas and procedures, or restlessly casting about for ideas" (p. 8). 

I'd like to put a different spin on the observation by the team that "there appears to have 
been a disproportionate emphasis on qualitative research in the Canadian e-learning 
research culture" (p. 3, p. 51). Contrary to the team's concern, I infer from this that the 
majority of educators are using emergent research paradigms (some might refer to as 
'free-ranging' or 'bizarre') in an attempt to conduct socially responsible research—the 
results of which provide useful guidance fore-learning developers and practitioners. 

In closing, I'd like to add that reviews of the literature are very difficult to do well, 
remarbably time consuming, easy for others to criticize, and one of the most valuable 
forms of publications when done well. Although I have cited areas that I consider 
problematic, this is a comprehensive and well constructed literature review. Educators in 
all sectors who are involved with e-learning should take the time to read the team's 
review. 
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